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Abstract 
This study leveraged data from a 40-school randomized controlled trial to understand the cost of 
coaching to support implementation of evidence-based programs (EBPs) through a multi-tiered 
system of supports for behavior (MTSS-B) model. Coach activity log data were utilized to 
generate the annual average, per school, costs of coaching of $8,198. The cost of school 
personnel time for coaching was estimated to be $3,028. Data on coach-rated administrator buy- 
in, school MTSS-B engagement, and implementation infrastructure and capacity were also 
collected and found to be associated with coaching activities. Notably, coaches did not spend 
significantly different amounts of time in schools using few EBPs relative to more EBPs, 
indicating some inefficiency in the use of coaches’ time. These findings highlight the often- 


overlooked resources needed to support EBP implementation in schools. 
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Tracking Time and Resources Associated with Systems Change and the Adoption of 
Evidence-Based Programs: The “Hidden Costs” of School-Based Coaching 

Over the past nearly two decades, research has demonstrated the positive impact that 
preventive interventions have on a range of social, emotional, behavioral, and mental health 
outcomes for youth when optimally implemented (O’Connell, Boat, & Warner, 2009). Given the 
enormous number of youth experiencing mental health, behavioral, and social emotional issues, 
there is a clear need to ensure that evidence-based programs (EBPs) are delivered to the 
population broadly and implemented with high fidelity (Fagan et al., 2019; O’Connell et al., 
2009). Unfortunately, the current use of EBPs among school practitioners is quite low (Hicks, 
Shahidullah, Carlson, & Palejwala, 2014; Stormont, Reinke, & Herman, 2011). Thus, a major 
challenge to adequately address the needs of youth in these settings is the scaling of EBPs in 
schools. For this to be accomplished, significant system changes are needed to provide 
professional development, resources to access and implement EBPs, and on-going support to 
implementers (Fagan et al., 2019). These system changes, however, may represent substantial 
costs to schools and districts. 

A growing body of implementation research highlights the critical role of ensuring that 
adequate implementation supports are in place to promote the implementation of EBPs 
(Domitrovich et al., 2008; Wandersman et al., 2008). Within schools, implementation support is 
often provided by a coach or consultant in the form of professional development as well as 
problem-solving to optimize fidelity and address barriers to implementation (Kraft, Blazar, & 
Hogan, 2018; Sprick, Knight, Reinke, & McKale, 2006). However, there has been limited 


examination of the costs associated with providing this type of implementation support and the 


opportunity costs borne by school staff who receive these types of coaching supports. Additional 
research is needed to better understand these often overlooked or “hidden” costs of coaching. 

To address these gaps, the current paper aimed to 1) summarize the amount of time spent 
on coaching activities and the cost allocations for systems-level coaching activities aimed at 
supporting the implementation of multi-tiered systems of support for behavior (MTSS-B); 2) 
estimate the costs associated with school-based personnel time spent engaging with the coach; 
and 3) examine the total time and cost allocation in relation to school engagement in the 
coaching and MTSS-B process and the level of implementation of new EBPs. These data were 
collected within the context of a school-level randomized controlled trial (RCT) in which 
coaching was provided to support MTSS-B. MTSS-B was meant to be a transformative approach 
to meeting the needs of youth, identifying when behavioral needs were unmet, and providing 
EBPs to address identified needs. As such, the coaching provided to school-level teams and 
school staff extended beyond the typical individual-focused coaching (e.g., of teachers or 
clinicians) to examine systems-level coaching. 

School-Based MTSS-B 

MTSS-B has been identified as a service delivery framework that can facilitate schools’ 
implementation of EBPs across the prevention continuum through the use of data-based decision 
making to address both school-wide intervention as well as students’ needs for targeted and 
intensive supports (Strein, Hoagwood, & Cohn, 2003). It is a complex process requiring 
transformative systemic change over 2 to 4 years (Eagle, Dowd-Eagle, Snyder, & Gibbons 
Holtzman, 2015). MTSS-B has been widely adopted as Positive Behavioral Interventions and 
Supports (PBIS), which focuses on improving school-level systems and procedures to promote 


positive student and teacher behavior change (Sugai & Horner, 2002, 2006). A growing body of 


school-based randomized trials and state-wide effectiveness studies indicate that the 
implementation of the universal (i.e., school-wide) elements of PBIS are associated with 
significant improvements in student behavioral, social emotional, and academic outcomes 
(Bradshaw, Koth, Thornton, & Leaf, 2009b; Bradshaw, Waasdorp, & Leaf, 2012; Horner, Sugai, 
& Anderson, 2010; Horner et al., 2009); school climate (Bradshaw, Koth, Bevans, Ialongo, & 
Leaf, 2008); decreased student need for additional behavioral supports (Bradshaw, Pas, 
Goldweber, Rosenberg, & Leaf, 2012); and improvements in student suspension, truancy, and 
academic proficiency rates (Lee & Gage, 2020; Pas, Ryoo, Musci, & Bradshaw, 2019). Despite 
these successes, many states and schools struggle to integrate targeted and intensive EBPs within 
this three-tiered framework (Barrett, Bradshaw, & Lewis-Palmer, 2008; Barrett, Eber, & Weist, 
2012; Pas et al., 2019) and require additional support and coaching to achieve high 
implementation fidelity (Fixsen & Blasé, 2008). Importantly, building systems-level coaching 
capacity has been identified as a challenge to the scaling of PBIS and other such EBPs state-wide 
(Fagan et al., 2019; Horner et al., 2014). 
Systems Coaching 

One important area for promoting the capacity of a school to successfully implement 
MTSS-B and adequately identify and address the needs of students (e.g., through EBP 
implementation) is consultation and coaching focused on systems change. Systems coaching 
within education has been defined as providing “dynamic support and facilitation to develop the 
capacity of school or district teams to implement [a multi-tiered system of support] (March & 
Gaunt, 2013, p.4). Systems coaching often involves a mix of activities, including organizing 


resources and the logistics of implementation, data-based decision making to examine fidelity 


and outcomes, and providing training and technical assistance while being mindful of the school 
context, culture, and climate (March, Castillo, Batsche, & Kincaid, 2016). 

The limited study of systems coaching to promote MTSS-B efforts has indicated the need 
for continuous coaching as well as alignment of educator beliefs with the MTSS-B framework as 
key facilitators of its successful implementation. For example, a study involving more than 60 
elementary schools indicated that educator beliefs about MTSS-B were associated with the 
implementation of MTSS-B practices. In addition, this study demonstrated that intervening with 
beliefs explicitly, by way of providing group professional development focused on supportive 
belief activities (e.g., watching video clips, having teachers reflect, sharing testimonials), both 
educator beliefs and MTSS-B implementation improved (Cook, Lyon, Kubergovic, Browning 
Wright, & Zhang, 2015). Similarly, when focused on MTSS for academics, a study in 31 schools 
indicated that the continuity of the coach within the role (1.e., having the same coach for a long 
duration) and supportive educator beliefs about data-based decision making were also significant 
predictors of implementation fidelity (March et al, 2016). 

Systems Coaching in the Current Study 

Within the current study, systems coaching explicitly targeted the capacity of school staff 
to identify behavioral and social emotional needs and select and implement targeted 
interventions for students at-risk for behavioral and social emotional concerns using a staged 
problem-solving approach (e.g., see Meyers, Meyers, Graybill, Proctor, & Huddleston, 2012). In 
these schools, MTSS-B was achieved through the PBIS framework and thus the primary purpose 
of the coaching was to support the PBIS team in collecting data, ensuring post-training follow 
through, and helping to tailor the framework to the specific school context (e.g., see Horner et 


al., 2014). The systems coaching provided in this study was meant to ensure the execution of 


activities outlined for PBIS coaching and to address implementation needs at every phase of 
implementation (i.e., from the initial exploration to program installation and implementation, 
with the ultimate goal of achieving full operation, innovation, and sustainability; Fixsen, Naoom, 
Blasé, Friedman, & Wallace, 2005). As such, coaches sought to ensure that evidence-based 
programs were well implemented across all three prevention tiers (i.e., universal, targeted, 
intensive; Fixsen et al., 2005). 
Estimating the Costs of Coaching 

Understanding the costs of systems coaching has some unique challenges in terms of 
economic evaluation. Specifically, cost analysis is used to determine the units and resources 
needed to implement an intervention, which most often include personnel, training, facilities, 
materials, and equipment (Levin, McEwan, Belfield, Bowden, & Shand, 2017). The challenge 
for systems coaching is that it is not a direct intervention with students, but rather is an indirect 
intervention that shifts the behaviors of adults to provide services to students in need of support 
(i.e., in the economics field, this is referred to as a service mediation intervention; Bowden, 
Shand, Belfield, Wang, & Levin, 2017). Therefore, it is critical to understand the exact activities 
that coaches engage in to support the implementation of EBPs (Crowley et al., 2018). These 
represent obvious costs, but also can hint at induced costs, which represent the costs that are 
incurred as a result of engagement with the coach (e.g., school personnel time spent engaging in 
the coaching) and represent opportunity costs, or the value of a good if it was used at its most 
productive alternative (Levin et al., 2017). Previous studies have noted that opportunity costs for 
teachers are not inconsequential. For example, a study of the costs of instructional coaching 
provided estimates ranging from approximately $8,000 to $28,000, depending on the number of 


teachers involved in coaching and their level of engagement (Knight, 2012). 


The accurate representation of cost is critical as it can dramatically affect the ratio of 
costs to benefits of the intervention (Bowden et al., 2017). Notably, improved implementation 
fidelity may result in more cost, but also improved benefits (Dane & Schneider, 1998; 
Domitrovich et al., 2008; Scott et al., 2019). For example, a study of the cost of school-level 
PBIS suggested that higher fidelity implementation was associated with additional costs, 
primarily driven by additional PBIS team meetings and supports (Bradshaw, Debnam, Player, 
Bowden, & Lindstrom Johnson, 2020). Another study of the actual time spent by PBIS coaches 
providing support for schools found that less direct and indirect support of schools (i.e., less than 
35 hours a month) was associated with improved PBIS outcomes (McCamish, Reynolds, 
Algozzine, & Cusumano, 2015). This may suggest a threshold effect, whereby costs increase in 
the earlier stages of implementation (e.g., Exploration, Installation as identified by Fixsen et al., 
2005), but then decrease as systems are in place and proficiency is reached. In economic 
analysis, this is discussed as efficiency, or how effectively inputs (1.e., support from coaches) are 
turned into outputs (1.e., implementation). Similarly, this study suggested that inefficient schools 
use resources but do not produce outputs (McCamish et al., 2015). 

Thus it becomes critical to both understand the “ingredients” of systems coaching that 
support the development and implementation of MTSS-B and how these ingredients and costs 
may differ depending on the level of buy-in, infrastructure, capacity, and school personnel 
engagement in the coaching and MTSS-B as well as the progress made (i.e., stage of) regarding 
EBP implementation. The primary cost driver in school-based interventions is often personnel 
(Belfield et al., 2015). Thus, methods such as time logs, which provide a detailed account for 
both intervention activities and time of coaches and school staff, represent best practices 


(Crowley et al., 2018). In this way, a cost analysis can both inform the resources needed to 


implement an intervention, as well as provide data relevant to the causal framework through 
which impacts on student social, emotional, and academic outcomes are obtained. Finally, this 
type of analysis can highlight opportunities for improved use of resources by suggesting school- 
level factors which relate to both the amount of resources used as well as the efficiency in which 
those resources translate to uptake of EBPs with fidelity. 
The Current Study 

Based on the gaps identified in the extant literature, our research aims were to: 1) 
summarize the amount of time spent on coaching activities and the cost allocations for systems- 
level coaching activities aimed at supporting the implementation of MTSS-B; 2) estimate the 
costs associated with school-based personnel time allocated to engaging with the coach; and 3) 
examine the total time and cost allocation in relation to school engagement in the coaching and 
MTSS-B process and the level of implementation of new EBPs. To address these aims, we 
leveraged two sources of data that are not commonly collected or reported in the literature 
regarding school interventions: coach time logs and salary data. To address the first two research 
questions, we calculated the costs of coaching provided to support improved implementation of 
MTSS-B, utilizing the ingredients method (Levin et al., 2017). Specifically, the coach activities 
were described, quantified, priced, and summed to create an aggregate school-level cost. As the 
focus of this paper was on the costs of coaching specifically, we examined the coaches’ time 
allocation and the opportunity costs of the school, as calculated using coach time logs and coach 
and school personnel salary data. To address the third research question, we combined the coach 
time logs of school visits with the coach ratings of EBP implementation and school engagement 
(i.e., four scales of buy-in, infrastructure for and capacity to implement, and school engagement 


with MTSS-B). Research on the link between cost and engagement and EBP implementation are 


10 


absent in extant literature but has major implications for the scalability of EBPs (O’Connell et 
al., 2009). 
Method 

Participants 

Data from this study were collected as part of a three-year, group RCT of the “Maryland 
Safe and Supportive Schools” or “MDS3” initiative. There were 40 middle schools across four 
school districts included in this study. Of these, 20 were randomly assigned to the intervention 
condition, received coaching support to scale MTSS-B to three tiers, and were the focus of this 
study. The middle schools ranged in size from small to large (IM = 789.55, SD = 145.23 students) 
and, on average, were comprised of a diverse student sample with regard to race/ethnicity (1.e., 
White students, on average, comprised 37.62% [SD = 23.35%] of student enrollment) and 
socioeconomic status (i.e., M = 40.30% [SD = 15.59%] students received free and reduced-price 
meals). These schools were experienced with school-wide PBIS, having been trained between 7 
to 13 years prior to the study beginning; a measure of fidelity to the school-wide, Tier 1 features 
using the School-wide Evaluation Tool or SET (Sugai, Lewis-Palmer, Todd, & Horner, 2001) 
reflected an average of high fidelity to the model, whereby only two schools scored below an 
80% (i.e., high fidelity) on this measure at baseline. Despite this longevity with school-wide 
PBIS, schools were not trained in or implementing EBPs at Tiers 2 and 3, which was 
demonstrated by the fact that half of the schools scored below 80% (i.e., high fidelity) at baseline 
on Individual Student Systems Evaluation Tool (ISSET; Lewis-Palmer, Todd, Horner, Sugai, & 
Sampson, 2005). 

Each of these 20 schools received implementation coaching support one day per week 


from an external, research grant-funded systems coach; there were six coaches in total who were 
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hired to support these schools (i.e., 4 full-time equivalent [FTE] positions designed to provide 1- 
day of coaching support a week per school). The six coaches were all female and were trained as 
educators or mental health providers (e.g., counseling, social work, school psychology); four had 
a master’s degree and two had a doctoral degree. 

Procedures 

Districts were approached to participate in the project by project leadership; the district 
agreed to participate and then provided support in recruiting schools for voluntary participation 
in the RCT. District-specific meetings were held with all interested school-based administrators 
to provide information about the trial (e.g., RCT design and all data collection procedures) and 
then administrators provided written consent for participation, where study procedures were 
outlined in writing. Matching procedures employing baseline demographic data were utilized for 
the randomization, to ensure balance across conditions (1.e., “intervention” and “comparison’’) 
within the four districts. 

All data reported here were collected prospectively from coaches (i.e., time logs and 
engagement ratings) or were administrative data (i.e., coach salaries). As this project did not 
include individual student- or staff-level identifiers, it was deemed exempt by the researchers’ 
Institutional Review Board. 

MTSS-B Intervention 

Schools in both conditions were provided access to a school climate data system, which 
allowed for annual collection of student and staff self-reported climate data, and real-time 
reporting of the survey results to inform school-based decisions regarding school climate, 
MTSS-B, and the implementation of EBPs (for additional details on the data system, see 


Bradshaw et al., 2014). Only the schools randomly assigned to the intervention condition were 
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provided with coaching to utilize these climate data to make data-based decisions regarding areas 
of need at Tiers 1, 2, and 3 as well as training in and materials and ongoing coaching for 
implementation of EBPs. 

Broadly speaking, coaches engaged in seven types of activities, including (1) relationship 
building to promote buy-in and readiness and to get to know the school climate, (2) participation 
in school-level meetings to gain information about school priorities and to help with alignment in 
priorities across faculty, (3) individual meetings with key stakeholders to ensure that beliefs 
aligned to MTSS and to build individual capacity for implementation, (4) coaching of individual 
teachers utilizing the Classroom Check-Up (i.e., CCU, Reinke, Herman, & Sprick, 2011) to 
ensure that positive behavioral supports were being implemented in classrooms, (5) supporting 
EBP adoption and implementation, (6) support for data collection and completion (e.g., fidelity 
forms), and (7) preparation of materials and planning for important meetings. For example, the 
systems coaches compiled information about the offered EBPs in a consumer-friendly manner 
for presentation to administrators and team members; this was done to ensure that school staff 
were well-informed of their options prior to choosing any program and as a preparatory activity 
to optimize schools’ EBP Exploration phase (i.e., the initial implementation phase, as defined by 
Fixsen et al., 2005). Wandersman and colleagues (2008) described this type of activity as 
distilling information. With regard to supporting all phases of implementation, when schools 
reached the phases of innovation or sustainability (i.e., the most advanced phases of 
implementation, as defined by Fixsen et al., 2005), coaches helped the schools to archive all 
related materials (i.e., ensuring that implementation could persist in the face of turnover) and 
discussed new ways to tailor implementation and measures in future years (see Horner et al., 


2014 for additional ways that coaching can support implementation at each stage). 
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Consistent with Fixsen and Blasé’s (2008) model of implementation drivers, coaches 
were carefully selected and trained by the research team to target all drivers (i.e., competency, 
leadership, and organizational). All coaches were supervised on a weekly basis by the lead and 
last (senior) author to ensure that coaches provided this comprehensive, ongoing support to 
schools. Examples of how coaches addressed these implementation drivers include: (1) building 
competence of teams and individuals through their meetings with these groups, (2) modeling 
adaptive leadership by building consensus, assessing and addressing the motivation of key 
stakeholders, and identifying key personnel within the building to implement EBPs during 
meetings, and (3) improving the organizational systems through their support of the collection 
and use of data to make decisions. The engagement around consensus and motivation aligned 
closely to addressing educator beliefs, which has been shown to be effective for promoting 
MTSS implementation in prior research (Cook et al., 2015). Finally, each coach was embedded 
into their assigned intervention schools for the three-year duration of the study, to maximize 
continuity and potentially impacts (March et al., 2016). 

Measures 

Daily coach time log. After each school visit, coaches completed an online log 
documenting the time they spent on each of the main coaching activities in the school (see Table 
1). This log was created based on the experience from previous research testing this systems- 
based coaching intervention in high schools (Bradshaw et al., 2014) and was designed to both 
understand the active ingredients of the intervention as well as to assess for coach and school 
time in a non-duplicated manner (Findorff, Wyman, Croghan, & Nyman, 2005). Throughout the 
study, logs were used to: (1) clearly articulate the expectations to coaches about how they would 


spend their time (i.e., to define the coaching intervention); (2) provide data-based formative 
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feedback to coaches about how time was being spent, (3) estimate costs, and (4) examine in 
implementation analyses. A codebook was developed for this data collection tool and was 
provided to all coaches. 

Coaches' activities outside of school sites (e.g., administrative work, research team 
meetings, research study data entry and analysis, and driving to and from schools) was not 
recorded in these logs. While these costs are a part of the total costs of the intervention and are 
reflected in the 40-hour work week of the coach (see Tables 2 and 3 for the “all hours” cost for 
coaches), they are not relevant to the research questions in this study. Additionally, while 
coaches were expected to be in schools for the entire school day, there was not an expectation 
that every minute in schools could be leveraged for systems change. 

There were eight broad categories of coaching activities captured in this log, which 
mapped to the above practice features: attendance and assistance provided at school-level 
meetings (i.e., providing the time spent at five distinctly-named meetings and a sixth entry for 
“other”; see Table 2), individual meetings with key stakeholders (i.e., the PBIS/MTSS-B Team 
leader, administrators, and other personnel), coaching of individual teachers, supporting EBPs 
with named programs and a section for “other” (see Table 2), relationship building (which 
included individually- and systems-based activities, parsed out into separate activities), data 
collection and completion (which was specific to the MTSS-B activities [e.g., fidelity forms] 
and not data collection for research purposes), preparation and planning, and other activities. 
For each activity, coaches were prompted to provide the number of minutes (rounded to the 
nearest 5 minutes; and only documented activities taking 5 minutes or more). Coaches were also 


trained that each activity could only be coded into one log category and were provided guidance 
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on the hierarchy of this coding. This was to ensure that no time was duplicated and thus costed 
twice. All logged activities were counted as coach time and costs (see Table 1). 

In addition, a designation was made for activities that involved teachers and school 
administrators that otherwise would not have occurred or were enhanced, due to the nature of the 
coaching. Specifically, although all schools across both conditions had received prior training in 
school-wide (Tier 1) positive behavioral supports, and thus should have been business as usual, 
school-wide Tier | team meetings were explicitly targeted by coaches as a time to promote 
systems changes. Further, there was an expectation and experience that the coaching increased 
the frequency of MTSS-B meetings and therefore was an induced cost; thus, teacher time was 
estimated as a cost for these meetings. Because of the importance of costing the personnel time, 
individual meetings were highest on the coding hierarchy, and were used for the estimation of 
teacher and administrator time. We were specifically interested in administrator time as a 
leadership cost. Therefore, if the coach met individually with an administrator, MTSS-B team 
lead, etc. about an EBP, they only coded the meeting and not the EBP support. This allowed for 
our estimate of school-personnel time but may have resulted in a slight underestimation of time 
spent on EBPs. Time logged as supporting EBPs, spent in individual relationship building, and a 
portion (i.e., 45%) of time spent in coaching individual teachers were also costed as teacher time. 
The latter was only a portion of estimated time because prior research has demonstrated that less 
than half of the time that a coach spends on a CCU case is also face-to-face time that a teacher 
dedicates (e.g., see Bradshaw et al., 2018; Pas, Larson, Reinke, Herman, & Bradshaw, 2016). 
While teachers and school administrators were not paid extra for the time in these activities, it 


represents an opportunity cost of engagement with the MTSS initiative (Levin, et al., 2017). 
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Coach ratings of engagement. Beginning halfway through the first year of the trial, 
coaches rated engagement three times per year (1.e., fall, winter, and spring). This measure was 
also created based on the experience of a previous research study of systems coaching (Bradshaw 
et al., 2014) and was intended to measure four dimensions of engagement with the MTSS-B 
initiative and coaching supports. Specifically, the a priori scales included the: (1) school’s 
PBIS/MTSS-B infrastructure and collaboration (7 items, e.g., “PBIS team demonstrates skills to 
independently own and direct the 3-tiered framework” and “PBIS team works collaboratively 
with me.”; Cronbach’s alpha or a = .88), (2) school’s capacity to implement EBPs (4 items, e.g., 
“School selected EBPs based on data” and “School personnel demonstrate capacity to implement 
EBPs soon after training”; a = .84), (3) the administrator’s buy-in and engagement (7 items, e.g., 
“An administrator actively supports the collaboration between me and the school staff’ and “An 
administrator allocates time and resources to implement Tier 1/Tier 2 and 3 supports” as two 
questions; a = .94), and (4) overall school engagement with the MTSS-B initiative (6 items, e.g., 
“School includes me in meetings and activities that are relevant to my work in the school” and 
“School is committed to ongoing data-driven decision-making”; a = .84). Coaches responded to 
each item on a 4-point Likert-type scale from strongly disagree to strongly agree. The items for 
each scale were averaged and higher scores reflected more engagement. 

Coach ratings of implementation of evidence-based programs. Beginning in the 
winter of 2015/2016 (i.e., halfway through the first year), coaches also rated the phase of 
implementation that the school had achieved for each of the offered EBPs including: whether 
schools implemented Tier 1 PBIS/MTSS-B, integrated restorative practices into MTSS-B, 
LifeSkills Training (Botvin, Griffin, & Nichols, 2006; Tier 1, universal drug prevention 


curriculum), Check & Connect (Sinclair, Christenson, & Thurlow, 2005; Tier 2-3 student 
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mentoring and engagement intervention), and Coping Power (Lochman & Wells, 2002, 2003; a 
Tier 2-3 group counseling intervention to address externalizing behaviors). See Bradshaw, 
Debnam, and colleagues (2009a) for additional information and support for the phased rating of 
implementation status by the coach. The coaches provided a rating of each school’s phases of 
implementation on a 7-point scale, which mapped onto the implementation stages outlined by 
Fixsen and colleagues (2005), including Exploration, Training, Installation, Initial 
Implementation, Full Implementation, Innovation, and Sustainability (also see Bradshaw, Pas, 
Debnam, & Lindstrom Johnson, 2020). Exploration and Training were collapsed into a “not 
implementing” category; installation and initial implementation were collapsed into an “initial 
implementation” category; and full implementation, innovation, and sustainability were 
collapsed into a “fully implementing” category. For this study, we focused on the final year of 
implementation (i.e., study year 3). 
Analyses 

Cost Estimation. Cost estimates correspond to the costing perspective of the coaches’ 
time, as such only annual recurring labor costs are included (1.e. fixed or indirect costs associated 
with coaching such as start-up training costs, supplies, equipment, overhead, and non-school 
costs are excluded). Costs are the summation of the costs of personnel listed from i=1 to =I. For 


66.299 
1 


each personnel “7” we determined its unit price P; and how many hours H; were required for the 
intervention, using the following equation: 


Equation | 


I 
Cost = > P;H; 
i=1 


where P is the expenditure on salary and wages and fringe benefits and H is the number of hours 


that each staff member allocated to MTSS-B support activities. Equation 1 was repeated for each 
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personnel category: coaches, classroom teachers, and school administrators and reported as 
annual estimates per school. All cost estimates were inflation adjusted to 2018 U.S. dollars using 
the Bureau of Labor Statistics’ Consumer Price Index Inflation Calculator (U.S. Bureau of Labor 
Statistics, 2018a). The source for hours allocated to coaching and coach labor costs were the 
daily time logs described earlier. Salary and fringe benefits data came from administrative 
records for coaches and from the Bureau of Labor Statistics for school staff (U.S. Bureau of 
Labor Statistics, 2018b, 2018c). 

Engagement and Costs Associations. Correlational analyses were conducted in SPSS 
between the eight main categories of coach activities and the four engagement scales. 
Specifically, we examined how the final (1.e., third) year’s coach logged time was associated 
with four coach-rated scales of engagement across all three years. This was intended to explore 
induced costs of the intervention, whereby working in schools that were more engaged would be 
associated with greater time and costs for the initiative. In addition, we examined the number of 
EBPs implemented in the final year of the study in relation to the amount of time the coaches 
logged in total during each year using one-way ANOVAs. These analyses allowed us to examine 
the mean differences between the number of hours coaches spent in schools over the course of 
the study in relation to final implementation status and were intended to explore efficiency by 
looking for threshold effects as well as both extremes (i.e., high efficiency or low inputs and high 
outputs and low efficiency/inefficiency as indicated by high inputs and low outputs). Because the 
goal was EBP adoption and implementation in the trial, final year implementation levels were of 
interest. 

Results 


Amount of Time Spent by Coaches and School Personnel 
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Based on the allocated distribution of time per school (i.e., 4 FTE for 20 schools), each 
school was allotted 402 possible hours of coach time (including summer). This was based on a 
40-hour work week for the coaches, considering paid vacation time and holidays. Of these hours, 
an average of 163 hours (SD = 43), or 41% of the total worked hours, were accounted for by 
logged activities of the coach. The range of coach hours spent within the school dedicated to the 
main eight activities’ categories varied from a minimum of 85 to a maximum of 232 per year, per 
school (see Table 2). 

The average number of hours that were logged as including school administrator and 
teacher time were 7 (SD = 28) and 59 (SD = 19) per school, respectively. As expected, out of the 
total school staff costs, most of the cost (83%) was attributed to teachers’ time. 

Cost of Time Spent by Coaches and School Personnel 

Labor cost parameters are provided in the bottom of Table 2. Using these values as well 
as the hour estimates, the annual average cost per school for coaching was $20,138 and for 
school staff was $3,028 (see Table 3). These values were obtained by multiplying the number of 
hours for the coach, teachers, and school administrators by their hourly wage including fringe 
benefits (see equation in Method section). Out of the total cost for coaching, the cost for the 
segment of coaching dedicated to school activities (i.e., logged hours) was, on average, $8,198 
per school. While the total cost was relatively constant over the years, the segment dedicated to 
school activities was slightly more expensive for year 2 than for years 1 and 3 (Table 3a). In 
accordance with the variation in total hours logged by coaches over the three years, the cost for 
school staff in Year 2 was higher than in Years | and 3 (see Table 3a). 


Distribution of Coaching Activities 
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When examining the costs per school, per activity, in all years, the cost of building 
relationships within the school was the highest cost (i.e., total for all years is $8,856) followed by 
the school-level meetings (total for all years is $5,412; see Table 3b). The total costs of school- 
level and individual meetings nearly matched the costs of the relationship building. The 
individual team meetings and preparation and planning each took about 12% of time and cost 
$2,952 over all years of the project. There was expected fluctuation in the costs for each activity 
over time, as the role of the coach was expected to shift. For example, coaches appeared to gain 
additional access to meetings and individuals in the second and third year. Similarly, EBPs were 
expected to be implemented in Years 2-3 (i.e., not the first year), thus, the time spent on this 
changed from 2% and $158 in Year 1 to 8% and 7% in the next two years. The preparation time 
decreased from Year 1 to Years 2-3 (i.e., from 17% in Year 1 to 8% and 10% in the next years). 
See Table 3b for more details. 

School Engagement and Costs 

Correlational analyses utilizing the activity data from the third year and all years of the 
school engagement data (i.e., three years * four scales) indicated clear patterns in the 
associations between engagement and coach time (see Table 4). Better coach ratings of 
PBIS/MTSS-B infrastructure and EBPs capacity in the second and third years as well as 
administrator engagement in all years were associated with more time spent in relationship 
building in the final year. Better school engagement with the broader MTSS-B initiative was 
associated with more time spent in school-level and individual meetings. Less engagement on all 
four scales was related to increased coach time spent on data completion. Poorer administrator 
engagement in the final year was associated with more coach preparation and planning time. 


Coach Costs and EBP Implementation 
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Coach ratings during the winter of the first year of the study indicated that 20% (n = 4 
schools) of the schools were not implementing Tier 1 PBIS, 15% (n = 3 schools) were in the 
initial stages of implementation, and 65% (n = 13 schools) were fully implementing. The focus 
of this first year was to ensure full Tier 1 implementation, assess areas that new EBPs could 
address, and begin to introduce and select EBPs for the second year. Therefore, not surprisingly, 
no schools were yet fully integrating restorative practices with PBIS and just three began initial 
implementation. During the first year, just one school each were fully implementing Check & 
Connect and Coping Power. No schools had implemented LifeSkills during the first year. 

With regard to end-of-study implementation, we examined the average of three ratings 
(i.e., fall, winter, and spring) of year 3 implementation for the four endorsed EBPs (i.e., Tier 1 
PBIS, integration of PBIS with restorative practice, Check & Connect, and Coping Power). 
LifeSkills was the one EBP that no school ever adopted. ANOVAs were conducted for the count 
of EBPs that schools either did not, only initially, or fully implemented (i.e., in three ANOVAs) 
to determine whether there were statistically significantly different amounts of time logged by 
coaches among these groups. The data indicated that there were no schools that were rated by a 
coach as not implementing any of these remaining EBPs (i.e., no counts of 4 for the “non- 
implementing” category). Similarly, no school implemented all four of the EBPs. Counts of 
EBPs that were not, initially, or fully implemented ranged from 0-3. On the other hand, there 
were nine schools each that initially or fully implemented two or three EBPs simultaneously 
during the final study year. 

Given that there were just 20 schools, as expected, there were a small number of schools 
in each cell and no statistically significant findings. In examining the means plots (see Figure 1), 


there are some indications of high efficiency and inefficiency, but not threshold effects. For 
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example, in Year 2, schools rated as not implementing 3 programs (i.e., non-implementers; see 
Figure 1, upper left) had the highest number of coach hours (top left of Figure 1). In other words, 
these were schools using the greatest amount of inputs (1.e., coaching resources) with no desired 
outputs (i.e., EBP implementation). Schools that were trying to initially implement three 
programs had the most logged hours in all years (top right of Figure 1); these were schools with a 
lot of inputs (i.e., coaching) while trying to produce the most outputs (1.e., launching the 
maximum number of EBPs). The number of programs was also notably associated with time 
spent when examining the full implementation categorization, such that schools not fully 
implementing any programs appeared to have lower hours than those with multiple programs 
being fully implemented (i.e., indicating a match between low inputs and outputs). A notable 
exception was in Year 1, where hours were lowest in schools fully implementing two programs, 
which was indicative of independent implementation of one, if not two, EBPs given that Year | 
predominantly focused on Tier 1 PBIS (bottom left of Figure 1). 
Discussion 

This study prospectively examined the cost of systems coaching that included coaching 
activities and elements consistent with prior conceptual, theoretical (e.g., Erchul, 2011; Gutkin & 
Curtis, 2009; Meyers et al., 2012), and empirical research (i.e., Cook et al., 2015; March et al., 
2016). We used the ingredients method (Levin et al., 2017), which represents best practice in the 
field of economic evaluation, as it includes costs of all components of the intervention and 
allows for an understanding of the cost of replication. Importantly, we used time logs that 
allowed for the prospective and detailed accounting of time spent by the coaches (Crowley et al., 
2018) and the associated costs as well as provided an understanding of the activities that the 


coach undertook during the three years including time spent with school personnel. 
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The results showed that the average annual cost of coaching activities in schools was 
$8,198 for coach time. Overall, coaches recorded that nearly half of their time was dedicated to 
school-based activities. While this can be seen as inefficiency, it is important for intervention 
designers and practitioners to understand that a full-time coaching position does not equate to 40 
hours a week of school contact and systems change; coaches cannot access the key stakeholders 
or engage in system change activities during all hours of the day and then there are other 
challenges like rescheduled school meetings and administrative tasks (e.g., email, trainings, etc.) 
that also take time. Notably, coaches spent about one-third of their logged time in relationship 
building, which allowed for needs assessment by the coach as well as readiness and capacity 
building (Wandersman et al., 2008) and may represent a “hidden cost” of coaching. An area of 
further exploration is whether or how the time (and costs) of building relationships would differ 
for coaches that were hired by the school system. On the one hand, the need for time spent in 
getting to know people and the organization would be lower for school system employees, but 
the readiness and capacity building would likely still be needed in some other form. Further, 
some estimates of time allocation for job-embedded coaches indicates that most of their time is 
spent in other activities, outside of the coaching they were hired to do (Fullan & Knight, 2011). 

Other specific logged activities data indicated that the intervention was implemented as 
expected and suggested important components of systems coaching. For example, coaches were 
advised to garner buy-in not only from the top (systems) level down, but from the individual 
(teacher) level up, which is reflected in the fact that they spent about one-tenth of their time 
coaching individual teachers. The second largest proportion of time was spent in meetings. Team 
meetings were targeted as the vehicle for executing MTSS-B and individual meetings allowed 


for building of motivation, buy-in, and skill sets of key stakeholders (i.e., the competency drivers 


24 


discussed by Fixsen and Blasé [2008]). Meetings with administrators addressed the importance 
of supporting multiple levels and leadership (Aarons et al., 2016; Domitrovich et al., 2008). 
Finally, separate time only allocated to EBPs (i.e., outside of meetings) was a lower time and 
cost allocation but is also captured in some of these other categories (e.g., meetings). 

Time logs also captured the important, but under-represented, cost in school-based 
interventions of school staff time, representing another “hidden” cost to coaching. It is common 
to assume this cost to be $0, as teachers and administrators do not receive any additional salary 
for these activities. However, it is important to capture the time they spend engaging with 
coaches as an opportunity cost (Levin et al., 2017). Specifically, this study suggests that nearly 
60 hours of teacher time and less than 10 hours of administrator time were “induced” by the 
coaching, which cost a total of $3,028 for school personnel time. As mentioned earlier, 
implementation science research not only highlights the importance of supporting personnel at 
multiple levels, including leadership, but leadership buy-in is also linked to sustainability 
(Aarons et al., 2016; Domitrovich et al., 2008). 

Finally, we examined EBP implementation. Notably, our findings indicated the greatest 
interest of schools in EBPs focused on Tiers 2/3 (i.e., Check & Connect and Coping Power) as 
well as restorative practices, which has gained traction nationwide in recent years (Gregory, 
Huang, Anyon, Greer, & Downing, 2018). During the study, schools shared with coaches that 
they were less interested in LifeSkills because they were not looking for a universal curriculum; 
reflected by no schools adopting it. We also utilized coach ratings of engagement and EBP 
implementation to examine the relationship between schools’ use of coach time and 
implementation success. Engagement was positively associated with coach time spent in 


relationship building and meetings and inversely associated with coach time spent on data 
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completion and preparation and planning. This supports that MTSS-B operates through a 
service-mediation perspective (Bowden et al., 2017) and suggests that coach face time may be 
critical for engagement. More time was also spent in relationship building in schools where 
coaches rated that the school had greater MTSS-B infrastructure and EBP capacity; this may 
have been because MTSS-B served as a connection and a common interest between the school 
personnel and coach, whereas schools less committed to MTSS-B may have been more reluctant 
to engage with the coach, due to avoidance, limited buy-in, or other work-related pressures 
precluding implementation. 

Although there was no statistically significant difference between the amount of time 
coaches spent in the schools and the number of EBPs that a school implemented, these analyses 
were likely under-powered given the small number of school units (i.e., 20). Visual examinations 
of these data suggested some resource inefficiency whereby schools that were not implementing 
any EBPs (i.e., indicated by a count of 3) also had a high number of hours logged by coaches. In 
other words, the goal was that coaching time (inputs) would translate into capacity built for EBP 
implementation (outputs), and yet coaches spent a lot of time in schools that never adopted an 
EBP. Coaches likely spent more time in these non-implementing schools to increase their 
engagement and implementation likelihood. While face time may be critical for engagement 
overall, this inefficiency suggests that face time with a coach does not necessarily translate into 
improved EBP implementation, possibly indicating that the approach of bring fixed innovations 
into schools worked for some, but not all, schools (Real & Poole, 2005), that alternative 
approaches may be needed altogether (Chorpita & Daleiden, 2010), or that a broader range of 
factors that are known to support implementation of EBPs in schools were needing to be 


addressed (Lyon et al., 2018). On the other hand, coaches also spent more time in schools 
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initially implementing three EBPs, demonstrating a more efficient use of the coaching inputs to 
promote EBP implementation. Some signs of schools achieving EBP implementation without 
additional coaching also emerged in examining the Year | data. 
Limitations 

This study focused on the specific activities of systems coaching and their associated 
costs. Given this focus on coaching, it does not represent the cost of the fully-implemented 
EBPs, which would have included additional administrative, staff time, and materials costs. It is 
important to consider this information in the context of the broader set of factors that support 
implementation of EBPs, such as leadership, the implementation climate, and commitment to 
implementation (Lyon et al., 2018). Unfortunately, we did not have data assessing engagement 
or implementation from teachers and administrators to minimize burden on school personnel. 
Additionally, due to the desire to ensure an accurate total cost and avoid double-counting time, 
the time spent on supporting EBPs may have been underestimated. An additional area that was 
not counted is email or brief and informal conversations (i.e., nothing less than 5 minutes was 
logged), which may have involved important relationship building or scheduling activities. 
Finally, it is important to understand a broader distribution of costs for school-based 
interventions, which might include support from the school district or state. 
Conclusions and Implications 

The systems coaching in this study sought to disrupt and improve school practices for 
service delivery to students at-risk of social, emotional, behavioral, and mental health concerns. 
Leveraging data from an RCT of MTSS-B, the findings indicated that the coach costs were a 
fraction of a full-time school personnel salary. Relationship building was a substantial portion of 


the logged coaching time, which is consistent with implementation research identifying buy-in as 


27 


a key driver behind implementation (Cook et al., 2015; March et al., 2016) and is not a cost 
frequently identified in economic evaluations. There are additional “hidden” school personnel 
costs to engage with the coaching, albeit a fraction of the cost of the coach. 

Costs, engagement, and EBP implementation are intricately related, and are of central 
interest in economic evaluations as it relates to cost efficiency. This becomes particularly 
relevant when considering a systems’ desire for coaching, which may both be a source of 
efficiency or inefficiency, but also plays into decision making regarding the desire for these 
services (e.g., a cost utility analysis) as do many other areas (e.g., feasibility, buy-in, and 
alignment to values; Hollands, Pan, & Escueta, 2019). Future work may inform the extent to 
which various coaching activities are functionally most cost effective in relation to optimized 
outcomes for the scale-up of EBPs in systems. These findings also provide greater awareness and 
transparency regarding some of the hidden costs of coaching that are often overlooked in 
planning EBP implementation and need to be considered when planning for EBP scale-up 


(Fagan et al., 2019). 
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Coach log components and costing of coach, teacher, and administrator time 


Coach Activities in Schools 


Coach 
Activities 


Teacher School Admin. 
Activities Activities 


Relationship building 
Systems 
Individuals 
School-level Meetings and Assistance*: 
PBIS meeting 
Pre-referral Team meeting 
SIT meeting 
Faculty meeting 
Grade Level meeting 
Other meeting 
Individual meetings 
PBIS lead 
Administrators 
Other 
Classroom check-up work with individual teachers 
(CCU) 
Supporting Evidence Based Programs (EBP) 
RP (Restorative Practices) 
LifeSkills Training 
Check & Connect 
Coping Power 
Other EBP 
Data 
Prep and Plan 
Other Activity 


KKK MK KKM OK 


~mK KK mM XK OK 


x 
x 

X (45%) 
x 


Note. The coach log excludes coaches' activities outside of school sites (e.g., administrative 
work, team meetings, data entry and analysis, driving to and from schools). The full time logged 
for each of the activities is counted to cost coach time; for the teachers and administrators, the 
X’s indicate the time that is also counted for their time. For coaching, 45% of the logged coach 
time is counted for teacher time, because research demonstrates that less than half of the time 
that a coach spends on a CCU case is also face-to-face time that a teacher dedicates (Pas et al., 


2016). 


Table 2. 


Cost Analysis Parameters for Coaches, Teachers, and Administrators. 
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Rereonnet Hours (” = 60 school observations; 20 school M hours SD Min Max 
3 years) (Annual) 
Principal/Administrator-hours ae 28 4 91 
Teacher hours 59 19 30 94 
Coach hours (Activities with schools) 163 43 85 232 
Coach hours (All activities) 402 - - - 
Labor costs (2018 USD) EGOS SD Min. Max. 
(Annual) 
Coach salary and wages $75,402 $5,080 $67,831 $81,856 
Teacher salary and wages $67,200 $10,655 $44,040 $101,340 
School administrator salary and wages $113,940 $16,305 $80,080 $155,180 
Fringe benefits (% of salary and wages) 34% 0.3% 31% 38% 


Note. These estimates include all time logged on the daily coach time-logs at the school sites, but excludes time spent off-site (e.g., 
doing administrative work, team meetings, emails, data entry and analysis, driving to and from schools). 
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Table 3. 
Personnel Annual Cost per School and Year (2018 U.S. Dollars) 
Personnel Annual Mean Percent of Cost Yr. 1 Yr. 2 Yr. 3 
Principal/Administrator $502 (17%) $450 $588 $469 
Teacher $2,526 (83%) $1,971 $3,059 $2,549 
School Staff $3,028 (100%) $2,420 $3,647 $3,018 
Coach (activities with schools)* $8,198 (41%) $7,918 $9,059 $7,623 
Coach (all other activities) $11,940 (59%) $12,328 $11,156 $12,330 
Coach Total $20,138 (100%) $20,246 $20,215 $19,954 
* Cost estimates include fringe benefits (33.54% of salary and wages). The total cost for coaching decreases slightly over time due to 


annual salary increments being lower than the annual inflation rate). 


Table 4. 


Cost and percent of time coaches spent in each of eight school activities 


Coaching activities with schools Mean % £Yr.l % #£Yr.2 % #£Yr.3 % 

Building Relationships $2,924 36% $2,851 36% $3,261 36% $2,668 35% 
School Meetings $1,776 22% $1,425 18% $1,902 21% $1,982 26% 
Prep and Plan $956 12% $1,346 17% $725 8% $762 10% 
Individual Meetings $956 12% $713 9% $1,178 13% $991 13% 
Coaching Individual Teachers $738 9% $713 9% $1,087 12% $457 6% 
Supporting EBPs $465 6% $158 2% $725 8% $534 T% 
Data Completion $246 3% $317 4% $181 2% $229 3% 
Other $137 2% $396 5% $0 0% $0 0% 


4] 


Table 5. 


Correlations between Engagement (all years) and Final (3") Year Coaching Time 
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Relationship School-level Individual Coaching Supporting Data Preparing & Other 
building meetings meetings teachers EBPs completion planning _activities 

Y1 PBIS/MTSS-B 0.428 0.436 0.160 -0.093 0.021 -.499° -0.397 0.057 
infrastructure 
Y2 PBIS/MTSS-B i hi 479° 0.288 -0.069 0.150 -.652"" -0.383 0.304 
infrastructure 
Y3 PBIS/MTSS-B 656 0.418 0.110 -0.382 0.239 -0.302 -0.349 0.373 
infrastructure 
Y1 EBP Capacity 0.214 0.347 0.134 0.018 0.102 -.537° -0.364 0.127 
Y2 EBP Capacity ATL" 0.433 -0.059 -0.296 0.429 -0.326 -0.338 0.098 
Y3 EBP Capacity 573” 0.276 -0.048 -0.409 570" -0.064 -0.298 0.224 
Y1 Admin engage 450° 0.437 0.221 -0.251 0.129 -.617" -0.272 0.054 
Y2 Admin engage AT4" 0.407 0.218 -0.227 0.215 -.462° -0.137 -0.018 
Y3 Admin engage 689°" 0.127 0.218 -0.357 0.285 -0.310 -.461" 0.294 
Y1 School engage 0.099 473° 466° 0.080 0.298 -.645™" 0.035 0.168 
Y2 School engage 0.144 0.347 464° 0.087 0.368 -.460° 0.239 0.074 
Y3 School engage 0.166 0.171 Sil 0.148 0.266 -0.264 0.106 0.126 


Note. Correlations are Spearman's rho. Columns represent the hours spent on each activity during the third (and final) year of the 
study. *p < 0.05, **p < 0.01 
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Figure I. Mean number of hours spent on coaching based on the number of EBPs that were not, were initially, and were fully 
implemented across three years. 
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